1. Field
Apparatuses and methods consistent with exemplary embodiments relate to image processing capable of performing an improved wide dynamic range (WDR) or high dynamic range (HDR) photographing functions.
2. Description of the Related Art
The use of a wide dynamic range (WDR) or high dynamic range (HDR) photographing function has recently increased. According to the WDR or HDR photographing function, a short-time exposure image (hereinafter, referred to as “short-exposure image”) and a long-time exposure image (hereinafter, referred to as “long-exposure image”) are continuously captured, and then, synthesized Thus, it is possible to obtain an image of a dynamic range beyond a maximum dynamic range that an image sensor is allowed to provide. The WDR or HDR photographing function is significantly effective for a high-contrast-ratio scene such as a backlight mode.
Criteria for selecting an image between a long-exposure image and a short-exposure image to be used for each pixel of a region usually include pixel value information and motion detection information. In a continuous image group, it is common that a pair of a short-exposure image and a long-exposure image are synthesized.
A scene, in which a camera slightly shakes, will be described. For example, a case where a fixed camera installed outdoors is slightly shaken by passing vehicles (or by wind), or a case where a portable camera is slightly shaken by a user's hand shake will be described. In such a scene, for a specific region of a WDR-synthesized image, selection of a short-exposure image and a long-exposure image is changed with respect to time. Thus, a difference of image quality between the short-exposure image and the long-exposure image may be seen as flicker.
A difference of image quality between the short-exposure image and the long-exposure image will be described. Generally, synthesizing WDR images is performed after a short-exposure image is normalized by an exposure ratio so that the short-exposure image has the same brightness as a long-exposure image. In a region in which a long-exposure image is saturated, it is impossible to make the short-exposure image and the long-exposure image have the same brightness. However, if the long-exposure image is not saturated, the brightness of the long-exposure image may match with the brightness of the normalized short-exposure image. However, it is difficult to make the two images have exactly the same image quality.
For example, if the normalization is performed based on the exposure ratio, an intended exposure ratio may not be equal to a ratio of an actually captured pixel value. This is because a setting accuracy of a shutter time is low or an exposure characteristic of an image sensor is affected. When the normalization of the short-exposure image is performed, a computational accuracy may also become a problem.
The short-exposure image generally has a lot of noise. Thus, even though the brightness matches, a difference in the amount of noise may be detected as flicker if the short-exposure image and the long-exposure image obtained by capturing the same subject are frequently changed.
The Patent Literatures below disclose technologies for reducing influence of external vibration on the WDR-synthesized image.
Japanese Patent Laid-Open Publication No. 2003-143484 (hereinafter, referred to as Patent Literature 1) discloses a technology that synthesizes images after absorbing misalignment of an entire screen by coordinate transformation. Even when a position of the short-exposure image is misaligned with a position of the long-exposure image due to vibration such as a camera shake, it is possible to synthesize the images regardless of such misalignment.
Japanese Patent Laid-Open Publication No. 2012-165213 (hereinafter, referred to as Patent Literature 2) discloses an algorithm in which, when determining which one of a short-exposure image and a long-exposure image is to be used, if a pixel value thereof is equal to or greater than a threshold value with reference to a long-exposure image, the short-exposure image is used.
Since the long-exposure image is saturated, the image selection is smoothly transitioned to generate a natural synthesized image in a high contrast region in which a region that uses the short-exposure image is adjacent to a region that is appropriately exposed and uses the long-exposure image. However, since a pixel value of the saturated long-exposure image is mixed, a color is lightened in a saturated portion of the high contrast region. Thus, the technology of Patent Literature 2 generates an image by enlarging a bright pixel value region of the long-exposure image by as much as a vibration amplitude, determines a selection signal with reference to the enlarged image, and enlarges a region using the short-exposure image.
Japanese Patent Laid-Open Publication No. 2011-135152 (hereinafter, referred to as Patent Literature 3) discloses a technology that performs bracket capturing with a short shutter time so as to maximally suppress a misalignment due to a vibration.
Japanese Patent Laid-Open Publication No. 2013-066142 (hereinafter, referred to as Patent Literature 4) discloses a technology that suppresses a synthesis artifact or a shaken image by using a short-exposure image in a motion region. In this technology, when a shaken long-exposure image is captured by movement of a subject, a short-exposure image having a small amount of shake components is used.
Using the technology described in Patent Literature 1 may be effective while observing one frame of WDR-synthesized image. However, when continuously observing WDR-synthesized images, the influence of vibration may not be disregarded. For example, when a position of the short-exposure image is shifted for alignment with reference to the long-exposure image, if the long-exposure image is influenced by the vibration, the influence of the vibration also remains in the synthesized image group. That is, according to the technology described in Patent Literature 1, selection of the short-exposure image and the long-exposure image is changed with respect to time by the vibration, and a part of the image may be seen as flicker.
In the technology described in Patent Literature 2, the saturated long-exposure image is not used for synthesis. Therefore, it may be possible to address generation of an unnatural color caused when a pixel value of the saturated long-exposure image is used for the synthesis. However, according to this technology, the position of a transition region of the short-exposure image and the long-exposure image is merely spatially moved. Therefore, for a specific region among temporally continuous WDR-synthesized images, the short-exposure image and the long-exposure image may be frequently changed by vibration.
According to the technology described in Patent Literature 3, influence of fast vibration may not be disregarded. In addition, even though it is assumed that the influence of vibration can be disregarded, noise caused by a short shutter time may be increased.
The technology described in Patent Literature 4 does not describe flicker that is included in a continuous WDR synthesis result when vibration exists. However, when entire vibration is detected as a motion, the short-exposure image is selected in a full screen and noise increases on the full screen. In addition, when vibration is not detected as a motion, an image to be used for the synthesis is selected from the short-exposure image and the long-exposure region according to a pixel value. Since the image used for the synthesis is temporally changed, flicker may appear.